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The present invention testes to a method and apparatus for deriving a three- 
dimensional representation from two or more two-dimens.onal images. 



As is weli known, in general such a 3D representation can only be generated ,f the 
fea res V points) of the images are correlated. Respective features of two 
Tl^ ^s are corrected if they are derived from (ie conjugate w,th he 
Ze Lure (el a point) of the object. ,f the positions ant I orientals o he 
camera a, which the overlapping images are acquired are known, then the 3D 
coord.nates of the object in the region of overlap can be determtned. assummg that 
the camera geometry (in particular its focal length) is known. 

15 Hu er al "Matchrag Point Features with ordered Geometric, Rigidity and Disparity 
odt a in sl circumstances (eg when the object is relatively simple or one o 

~:; n e tr:ji - «- «— — « — — 

an embodiment of the method and apparatus of the present mvenuon. 

of features. (Stereoscopic camera arrangements in which the cameras are fixed are 
ignored for the purposes of the present discussion). 

EP-A-782,100 discloses a method and apparatus in the first f^™^ * 

built up from 2D images acquired by a digital camera. Both the position 
orientation of the camera are monitored. The hardware required to achieve 

expected to be expensive. 

A number of research papers have addressed the more difficult question of deriving 
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• r ™, .he correlated features of the images, as follows, 
the correct orientatton from the correlat 

■ , Algebraic Formulation of the problem of Relattve 
i> E H Thompson "A rafonal Algebratc J ^ ^ ^ ou , 

Orientation" Photogrammemc Record v ^ (o fte COItect 

mathematics, procedure for _»» a „ ^ solutio „ of five 

Richard Hartley - a, "Stereo "^^ZZ^to^ 

Computer Vision and Pattern **">%£^Jv»*- » •* - «" 
camera matrices define the came a onenU. caUbraKd (ie the internal 

camera parameters such as focal »"*^ f £^ ca „ be found from the matched 
parameters are known) then the camera _ main ^ Jf ^ cameras 

U ^--^^""rU - be used to derive the 
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^aii«^ iv * ** 

■ • „f Relative Camera Positions for Uncalibraled 
Bi) Richard HarUey Vo. 588, 1992 pp 579-587 ,s a 

M Hartley - - "S^ETS^^r 

be acuuired by = easily . Als o the cameras do not 

The remaining points can tnen oe 
need to be calibrated. 

a the intensive computation 
A disadvantage of the above ™>»— „ is computati on involves ,h. 

Quired to process all the correlated P°'«^ {mm mulliple points and is 

- = P rTosmtns ,0,.- -* ~ 
de— »y - large number of processed po.nt, 

„ has been shown tha, .n ^^^^ ST 
and orientation) of the camera(s) ca . b found ; ^ ^ 

However there are specia > ' „di„g to the pairs of 

is a situation in which all eight pom 

corelated points in the images are cohnear. 
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lropMtantly , another situation in As 
when .he camera orientals (m the ^ nc camera separa ,,on in such a 

wi „ be apparent from Rgure 3 (**-^^ pairs of points are correlates, 
situation is qutte indetermmate, no ma ter how many P ^ ^ . mages 

640 X 480 pixels) it will be apparent that tf ^the cam ^ ^ and 

different then there will be constde able ^unc rmnty ^ , f fte ^ _ 

hence in the 3D reconstruct^ of he object. ^ 

alleviate at least some 
An object of the present inventon is to overco m ^ ^ 
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. , method of deriving a 3D 
Accordingly, in one aspect the ^«;* S lated over ,apping 2D images of 
representation of at leas, part of an object from co ^ ^ ^ , he 

^object acquired from different spaced *££"£\ lel m e method 

separation between the v.ewpotnts no. bem P > ^ a 3D 

"wpoints a. which the 2D images were acqu.red. 
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V 1^ f v ** 

for derivin° a 3D 
Th e invention also provides ««-<J^f ^ e 3dler.appi»g 2D 
representation of a. leas. par. of an object from tQ [he object . the 

J object acqu,red from different JP^™^ ^ to disitall y process 
apparatus comprising image process.ng means w mc simu , a ,ed3D space m 

^2D images to form a 3D ^T^t ^^s of .he respecive 
dependence upon both the ^f^^Z^ representative of the 
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use the scaling variable is preferab.y enured by a user. 

is a and the partial 3D reconstruction is 
For example if ft. ^T^ZT^ P-meters as the camera*, 

generated by virtual projectors with th «m P ^ scatag faclor couW be 
and having a separation of a' in {actor of M and thereby generate 

a/a' to magnify the partial 3C 'r^^. as fte object. Such a partial 3D 
a partial 3D reconstruction having the am 3D reconsmlc „ ons 

Construction will be able to be fined to other 1 any value of 

generated similarly from ^ ^ instructions to be fitted together 
T b 5 ST " — t applied by the user dunng a process of 

■ n « the reference frame of the object) at 
ln ft.se embodiments the camera option. ^ refe ^ _ ^ 

each of the two - preferably generated by virtual 

Agrees) and ^rT— - the camera(s) and having optica, centres on 

rt» <ieo of acquiring the overlapping 2D 
Pr eferab.y the method f»*.^ ^ , he ^ between the different 

constrained. 

a fixed slide so as to move 
For example the camera could be — J"^ moveme „, along two axes 
2 5 lr a„sv.,se,y to its optical axis ( so th at ■ on nia « ^ ^ ^ ^ 



a tripod (so that its movement in 
axis are constrained). 
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However in a preferred embodiment the camera is hand-held. 

h, measured with an inertial sensor eg a 
Optionally, the camera orientation can b m ^ as disclos ed in 

■ a aft^r acauiring a first image from 
Preferably the orientation of the om V other viewpoint so as 
rrrrJ2T=i - referee frame of the object when 
acquiring the second image. 
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relative to the orient a. the the coLated points of the first 

projections in the image o He ^ h preferab , 

image and the correspond pomu ot ]n ^ , 

trusted »«" '»e above projections are parallel. 

• „>„ relates to a method and apparatus for deriving a 
The present invention also relates to ^ an , mage 

representation of the t hree-dime„s,ona. 3D) ^ ^ ^ 

radiation onto the objec, surface. 1 ^ ^ nQt only 

which obeys the laws of optics. 

w ,t cnrface can be obtained by projecting 
,„ principle the 3D shape of part of an urface ~ ^ ^ rf 

distribution of the structured light is known. 

- 5 ^ 1 ;« n Surface Solution Using 

One such arrangement is shown by Hu ^ % No 4 pp 

— ^hoXT r=g- - — e over stereoscopic 
imaging techniques. 

30 

ln a second aspect the invention ^^^t^X^ 
of at least par, of an objec, from a 2D ^ma e ^ a 2D 

nominating the object with ■^«»« l ^ ) , * ll|e w i,„ rays of the structured 
image of the illuminated object, ' ™. ^ ,„ form a 3D representation 

» optica! radtatio, and digi "^'^^upon both the correlation and 
which extends in a emulated 3D sp <mu rf , he separauon between 

: — * — d - ,he v,ewpo 
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at which the 2D image is acquired. 

this aspect the invention also provides ^^^^^^^ 
3D representation of a. least par. of an object t'™*™'™^ projected from a 
object, the object being illuminated w.th structured optic, £ "on P J 
location spaced apart from the viewpo.n. at ^^Z^^m digital 
im a 8 e being corrected I a simulated 

processing means arranged to form a 3D "P""™" 1 riab , lhe sca i ing 

r tu^pLX" project^ and the viewpoint a. wh,ch the 20 .mage .s 



acquired. 
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not need to be known in the apparatus and method , of ^the «c=n p 

dear analogy between the camera-camera ,„ each case, a 

and the camera-projector arrangement employed ,n he second asp ^ 

3D representation can be derived from t e ,„ ers^uon o two OJ e ^ 

case representing the respecve pencls of camera ™£ »" 

representing the respective pencils of projector rays and camera rays. 

The catibration image can for sample be of the pr^ontf .hestiu ctured optic, 
radiation onto a calibration surface or can for example be a furth J 
obtained after moving the object relative to the Iadiation . 
object image and the projector means used to project the structured p 

Preferably the first and second projections ^^^^ 
the structured optical radiation and a perspective en« ^ 
(eg the optica, centre of ^ d a seline from two or 

reconstruction processing means being arranged to ' ^ and , 3 

more pairs of correlated features. Th.s feature ,s tllustrated 
discussed in detail below. 

„ : c nrranoed to generate 

ln 0 „e embodiment the image processing - ^ ihe 

correspondences between two or more cahbration .mag« ^d 

spacing between origins of the firs, and ^™^ZZ« or s.ored metric 
the correspondences of the two or more cahbration, ma ^ J 
information associated with the calibration .mages. Th,s feature 
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Figure 15, discussed in detail below. 

spacing between ,he ongms of .he «^ caUbration image 

variable. 

of a „ ob.ec. torn an ^^^J^.. im age of .be projection of 

15 method comprising the steps of: 

„ coding a. ieas. one caiibration image with ,be object image and optional with 

a further calibration image; 
» ii, sim-ing a fir* flection of .he objec. image an, a second projection of ,he 

structured optical radiation, and 

iii, deriv.ng said 3D representation from the mutna, intersections of the f,r, and 

os second projections. 

■ 3D r r :Tr:;J 0 o„,o ,He objec, ^ ^ , 

projecuon of structured ° f ^ omo a 

processing means —^^7^3^ — 

rr^T^-— - — :::crr 

» object image and a second ^^^Z^Z, - — 
two of the correlated images and to derive said iu P 
intersections of the first and second projections. 
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Preferred feau.es of .he invemion are defined in .he dependen. data, 

- ^rr> thP viewDoint at which the 2D image ^^4 
aptut from * radiation , the appara .„s compns.ng d,g«al 
correlated w..h rays of .he stt ' wUch extends in a simulated 

processing means arranged o f«« ^ — ^ ^ ^ 

3D TltC^<^^-- » the locat,on from wteh ,he 

ZZSZZZL* is Pieced and ,e viewpotn. a. which *. *D .ma g e , 



acquired. 



0 



25 



30 



35 



Thls aspec. of ,e in«n.io„ is _d in £ 
ca.ibra.ion /^^J^^D -*J^ » * * ™ s 

:::::::: ^ ^ * - * — * - — — - 

graphics and animation. 

Preferably * apparams is — S^T^ 

* tho further 3D representation by manipulations in a simuiaie 
and the further 3D P ^ further compnsing 

involving one or more ol rotation a ffmain ; na discrepancies between 

scaling means arranged to reduce or ehmtna.e any — <* * - P ^ 
the 3D reconstructions by scaling one 3D reconstructs relauve 

least one axis. 

Preferabiy the apparatus is arranged to 

simuhaneously and to manipulate them „ emulated 3D space 

commands entered by a user. 

,„ one embodiment the apparatus is arranged .o perform the mansions of ,he 3D 
reconstructions under the control of a computer pointing dev.ce. 

Preferably the apparatus inCudes means for combining .wo or more 3D 
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repressions and means for adjuring .he relative scaling of .he represents to 
enable them to fit each other. 

In other embodiments .he variable will correct for one or more distortions of the 

3D ration ^ * *. * : ^ 

which as shown below in connection with Figures 8 to 1 1 can arise irom 
^itnTng of the virtual projectors eg a misalignment rdative to the camera 

viewpoints. 

in certain embodiments the partial 3D reconstruction will be distorted by delibarately 
misanglTone or both the virtual projectors relative to the camera viewpoints or 
camera and projector viewpoints. 

Such a feature is useful in the fields of design, graphics and animation. 

Some difference in orientation can be tolerated and the resulting distortion in the 3D 
reconstruction subsequently corrected, as will be shown below. 

The smaller the difference in orientation, the smaller the distortion in the 3D 
lecons'cL. Ideally the orientation of the camera is maintained unchanged 
20 between the two viewpoints. 

Preferably the angle subtended by a pair of correlated features a. the 
feature of .he object is 90 degrees ± 30 degrees (more preferably ± 10 decrees), 
del the subtended angle is exactly *> degree, This feature enabies any distortion 
resulting from a slight change in orientation to be corrected more accurately . 
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25 



overlapping images can be fitted together. 

30 To the extent that the reconstruct, on is distorted then ^^^J^ 
distortion parallel to the image plane and for curvature of field can be applied 
order to enable the 3D reconstruction to be fitted. 

35 Preferably a view of the reconstruction in the simulated 3D space is displayed eg on 
a tlfn Ld the variable is varied by the user in response to the view displayed on 



screen. 
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Preferably the image processing means is arranged to generate said correspondences 
of said images by comparing local radiometric distributions of satd tmages. 

,n a related aspect the invention provides a method of deriving a 3D representation 
of a. least part of an object from a 2D image thereof, compristng the steps of 
illuminating the object with structured projected optical radiation, acomnng a 2D 

5 imaTe of the illuminated object, correlating the 2D image with rays of the structured 
optical radiation, and digitally processing the 2D image to form a 3D reconstruction 
which extends in a simulated 3D space in dependence upon both the correlate and 
a scaling variable, the scaling variable being representative of the separation 

,0 between a location from which the structured optical radiation is projected and the 
viewpoint at which the 2D image is acquired. 

Suitable algorithms for correlating (generating co^espondences between) 
overlapping images are already known - eg Omen's algonthm (see 0»,» W 
15 -Adaptive leas, squares correlation: a powerful image ma.chtng techn.que S Afi J o) 
nJma^r,. rem*, sensing «n* Canopy Vol 14 No 3 (1985) and Gruen 
A W and Baltsavias, E P "High precision image matching for dig,tal terram model 
generation" ,* ArC pno^e.ry Vo, 25 No 3 (1986) 

^ion-growing" modification thereto which is descnbed m Otto and Chau 
20 ..Re^growmg algorithm for matching tetra.n images" too* «i ^ 
Compos Vol 7 No 2 May 1989 p83, all of which are incorporated herem by 

reference. 

» Essentially, Omen's algorithm is a„ adaptive least squares correlation algorithm in 
which two image patches of typically .5 x 15 to 30 X 30 p.xels are collated 0 
selected tm larger left and right images in such a manner as to gtve he most 
consistent match between patches) by allowing an affine geometnc : d_ 
between coordinates in the images (ie stretching or compresston tn whtch ongtn y 
30 parallel lines remain parallel in the transformation) and allow.ng an add.ttve 
Liometric distortion between the grey levels of the pixels in the .mage pa«hes 
-eneratin. an over-cons.rained set of linear equations representing .he dtscrepanc.es 
"between .he collared pixels and finding a leas, squares solution which rn.ntm.ses 
35 the discrepancies. 

The Gruen algonthm is essentially an iterative algorithm and requires a reasonable 
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- • for the correlation to be fed in before it will converge to the correct 
approximation for the correlation approximate 

s^tion. The OUo and Chau ^'^^To^. ut „ises On-'. 
m atch between a point in one ,mage and a pom, tn ^ ^ 

algorithm to produce a more accurate ma ^ b and £ , n ^ 
radiometric distortion parameters and uses h rf (he initia , 

approximate matches for points m the re g o o he ^ ^ 
matching point. The neighbour potnts are selec ^ 
points on a grid having a grid spacng of eg :> or 10 ptxels 
Gruen's algorithm for every pixel. 

Hu „ „ -Matching Point features ^^^^Z 
~r -"ZC^Js cited there, dtscloses further 
methods for correlating features of overlapping images. 



follows: 
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0 the additive radiometric shift employed in the a,gori.hm can be dispensed with; 

U) if dun „g successive iteration, a >^^J^^Z^ 
certain amount (eg 3 pixels) per .terauon then ,. ts 

should be rejected; 

„ during ,e growing of a matched region ^^^^ 
contrast at a. leas, three of the four stdes of the ^ » ^ ^ 
sufficient data for a stab.e convergence contrast) to 

make the algorithm configurable to enable the parameters («„ 
be optimised for different environments, and 

, v) i„ order to o.uantify <ne ^ - - — ~ " ^ 
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• 12 i odd ooint in the starting image by 
be e» found useful to derive the ongma ^ ^ (ie reversing the 

applying the algorithm «o . e — p,nt , ^ beween ^ . nai g[id poin( 

it will be computationally inefficient) 
It is no, necessary (and indeed in many ^ viewpoints 0 f the camera(s> 

,„ correlate all possible features prior to deter™ . ^ c0rIelatl0n s 
re , ati ve to the object. U wil, usually * (he viewpoints have been 

0 between features of the of correlated features, by searching 



determination 
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Ouce the correlations and ^j££%%£Z 

the location of the object feature. 

A ccording,y. in another aspect the — ^eT^ ^ 
.construction o, an object ^JsD space from aligned virtual 

OMhK for oenerating a 3D 
ln lh is aspect the invention also ^J^ ^'™™*™^ 

— — of : £££££ — — r inl ° s,mu,a,ed 

th em) is less than 45 degrees, 7" p f * r «es. Desirably this angle is less 
preferably less than 20 degrees, eg .ess than* . ^ of 

, 5 L 5 degrees, eg less than one and also facHitates the 

° th e object to be captured ~ * cal «„«res - - — » « " 
oetermination of the ^ne — P ^ ^ images . 

different viewpoints as wen as 
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Preferably ft. origm of each ^^Z^jLl^S^Z 

z^^^^ z - - ,n a sca,ed panial 3D 

reconstruction of the object. 

™.t is entered by the user and applied to the 
ln one embodiment a distort,*, parameter enter y ^ ^ 

3D reconstruction. For inject to form an initia. 3D 

their nominai «^J^J?JXm 3D reconstruction can be rotated 
reconstruction in emulated 3D space, tn clion wWc h are generated 

0 whilst constraining the features of the .mual ^ K he 

from the in.ersecung projections of ima ' es there b y forming a 

on the projections of those features rom one of ft. 2D , ^ ^ be 

further 3D reconstruction. As ,s .Uustrated ,n R u 9 ^ ^ ^ ^ 

used to generate a rotation and constrain, correct for 

15 of, the actual object ™ parallel ism of the initial 3D recons.ruct.on 

the lateral dis.ort.on caused by .he lack ot p projectors, 
and the 3D reconstruction generated from correctly al.„n 

20 

effect. 

sssssssssssfsbsss 

aTsocia.ed pairs of correlated points, however acoutred. 

Preferably the — are ^Z^^^ 

^- f ^ wedf ;:;::;: of : viewpoints, «. — - ** 

T ,,s preferred feature of the — is ^J^£?£Z 
whi ch shows the derivauon of <*J^^£ me thW project from 
projections from mree pa,rs of . nlersecK lhe poinl VP already 

the third pair of correlated fea.u es ^P3 and K ^ ^ 

defined by the other two projecuons mters cno C ^ ^ ^ 

straigh. line joining the viewpomts. Hence th.s l.ne can 
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of correal features if tore is no change in camera orientation. However greater 
accuracy wil, be obtained if more than two pairs of correlated features are processed. 

Preferably said calculation is performed on fewer than one thousand pairs of 
5 IZJi features, more preferably fewer than one hundred pairs of corre a«d 
features, desirably fewer than fifty pairs of correlated features eg e.ght or fewer 
pairs.. For example the calculation can be performed on four, three or two patrs of 

correlated points. 

>0 particularly when the viewpoints are parallel or nearly parallel (eg within 10 degrees 
of each other) the above preferred features result in a greater or lesser degree of 

economy of processing. 

Further preferred features of the invent.on are defined in the dependent claims. 

15 

Preferred embodiments of the invention are described below by way of example only 
with reference to Figures 1 to 20 of the accompanying drawings, wherem: 

20 Figure 1 is a diagrammatic view of one apparatus in accordance with the two- 

camera aspects of the invention; 

Rgure 2 is a flow diagram of one method in accordance with the two-camera aspects 

of the invention; 

R.ure 3 is a ray diagram showing the relationship between the object, camera and 
proctor viewpoints, virtual projector viewpoints and partial 3D reconstructs .n 
one embodiment of the invention ; 

30 Fioure 4 is a ray diagram in 3D showing a derivation in accordance with one aspect 
„f°,he invention of the direction of movement of the camera from the acqun-ed 
images in the method of Figure 2 and apparatus of Figure 1 ; 

R„ure 5 is a ray diagram in 3D showing a derivation of the direction of movement of 
the camera from the acquired images in the method of F.gure 2 and apparatus o 
Hgn" 1 in the special case in which the camera does not move relative to the object 

in the Z direction; 



35 




PCT/GB99/01556 



WO 99/60525 



15 



Figure 6 is a diagram showing the movement of the image of the object in the image 
plane I of Figure 5; 

5 Figure 7 is a flow diagram summarising the image processing steps utilised in the 
method of Figure 2 and the apparatus of Figure 1 ; 

Figure 8 is a 2D ray diagram illustrating the curvature of Held resulting from 
misalignment of one virtual projector relative to the other in the apparatus of Figure 
10 1 and method of Figure 2; 

Figure 9 is a 2D ray diagram illustrating correction of d.stortion of the partial 3D 
reconstruction in an embodiment of the invention; 

15 Figure 10A is a 2D ray diagram illustrating the curvature of field resulting from a 
misalignment of a virtual projector by 5 degrees; 

Figure 10B is a ray diagram illustrating the curvature of field resulting from a 
20 misalignment of a virtual projector by 10 degrees; 

Figure 10C is a ray diagram illustrating the curvature of field resulting from a 
misalignment of a virtual projector by 15 degrees; 

25 Figure 1 1 is a plot of curvature of field: misalignment in the arrangements of Figures 
10A to 10C; 

Figure 12 is a schematic representation of one apparatus in accordance with 
3Q projector-camera aspects of the invention; 

Figure 13 is a sketch perspective ray diagram showing one optical arrangement of 

the apparatus of Figure 12; 

35 Figure 14 shows an object image and a calibration image acquired by the apparatus 
ofFigures 12 and 13 and the correlation of their features; 

Figure 15 is a sketch perspective ray diagram showing a variant of Figure 13 in 
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which .wo reference surfaces are used to loca.e the camera and projector of the 
apparatus of Figures 12 on the baseline connecting their respective perspec.tve 



centres; 



Boure 16 is a flow diagram illustrating a method of operation of the apparatus of 
Fibres 12 and 13 in accordance with a projector-camera aspect of the invenUon; 

Route 17 is a screenshot illustrating the fitting together of two 3D surface portions 
oAhe object using the apparatus of Figures 1 and 2 or the apparatus of Figure 12; 

0 

Figure 18 is further screenshot showing the scaling of the resulting composite 3D 
surface portion along vertical and horizontal axes; 

Figure 19 is a further screenshot showing the scaling of intersecting 3D surface 
portions to fit each other, and 

Figure 20 is a screenshot showing a user interface provided by the apparatus of 
figures » and 2 or the apparatus of Bgure 12 for manipulating the .mages and 3D 

20 surface portions. 

Referring to Figure 1, the apparatus comprises a personal computer 4 (eg a 
P .n,i„m® PC) havin- conventional CPU. ROM, RAM and a hard drive and a 
Tin a" an" ,nput port to a dtgtta, camera 1 as we„ as a video output pon 
* connected to a screen 5 and conventional input ports connected to a keyboard and 
mouse 6 or other pointing device. The hard drive is loaded w„» conventional 
operating system such as W.ndows®95 and software: 

a) to display images acquired by the camera 1; 

b) to correlate points in overlapping regions of images input from the' camera 1 ; 

c) to derive the line of movement of the camera from the acquired images; 

35 d) to proiec. images acquired by the camera into a simulated 3D space from virtual 
project located on the line of movement at a separation selected (wuh the 
keyboard or pointing device) by the user (thereby creafng a parttal 3D 



PCT/GB99/01556 
WO 99/60525 



17 

reconstruction); 
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e) to scale the partial 3D reconstruction along one or more axes and combine such 
partial 3D reconstructions as illustrated in Figures 12 and 13, and 

0 to apply or compensate for lateral distortion and curvature of field as illustrated in 
Figure 9 and Figures 1 OA to 10C. 

The software to carry out function a) can be any suitable graphics program and the 
software to carry out function b) can be based on the algorithms disclosed m Hu et al 
"Matchino Point Features with ordered Geometric, Rigidity and Disparity 
Constrain*" IEEE Transactions on Pattern Analysis and Machine Intelligence Vol 
16 No 10, 1994 PP 1041-1049 (and references cited therein). One suitable algorithm 
is the Gruens algorithm. 



Referring to Figure 1. the camera CM, a Pulnix M-9701 progressive scan digUal 
camera, which may be hand-held and carrying a 3-axis vibratory gyroscope G with 
associated filtering circuitry (a Kalman filter is presently preferred) and integrating 
, 0 circuitry to generate and display on screen 3-axis orientation signals or (as shown at 
" 1') may be mounted on a tripod T or other support, for example, defines a first set of 
axes x y, z in its initial viewpoint (ie position and orientation) and is used to acquire 
an imaoe of the object 3 and to display the image on screen 5. The origin of the x y z 
coordinate system is taken to be the optical centre of the camera lens. The camera is 
25 then moved an arbitrary distance to a new position CM' and a second image of 
object 3 is acquired with the orientation of the camera (relative to the x y z 
coordinate system) maintained unchanged. This unchanged orientation can be 
achieved with the aid of a suitable support if necessary. A method of checking for 
chances in orientation, based on the convergence of the projections from 
30 corresponding points of the two images in the image plane I of the camera, will 
subsequently be described with reference to Figure 4. Only a few (eg 3, 4, 5 or up to 
ec 10) pairs of correlated points need to be found for this purpose and can be derived 
visually by the user from the images displayed on screen or by the software m 
computer 4. As will become apparent from Figure 4, the camera movement (ie the 
line in the x y z coordinate system joining the optical centre of the camera lens in 
its two positions) can also be found at his stage. In some cases however in winch a 
highly accurate reconstruction is not required, the direction of camera movement can 
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be estimated by the user. 
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The remaini„ g co— g po.nts i. -he - ^^^^ 

— « <""" * TSZ^ZSZZ in - image 
^ '^'^TJ lTTL o,„er image, and a partial 3D 
corresponding to a pomt of m.eres .» . ^ 

reconstruction of the object 3 ,n s,mu!ated 3D space s 

-7 ; h ; p r :i e t fiilyh^ or *. **. « 

,S The compu e r 4 ■„ p , og rammed .0 display the resu,tin g 3D reconstn.ct.on 

P 1 The usl can vary the parameter interactively in order to amve at a 
on screen and the user y P which ^ a desired 

paniai 3D reconstrucfon ,n , e ^ ^ » or compressed in 

relationship to the actual object 3. Typ.ca ly tins ^ 
,he direction of movement between positions CM and CM , 



object. 



20 



„ „• ™ the accuracy with which the camera orientation is ma.nta.ned m the 
Depe„d.ng on he accuracy ^ Q) and ^ 

,wo pos.„ons (eg w.th he a d of ^ ^ ^ 

„ of . e e— wW£h ^ ^ conected a( lUs sBge . The 

distortions in the partial ^ re^i ft<w;t ; on of such distortion 

correction of such distortion (and indeed the M '^^^r^s 9 and 
when this is desired) will be described subsequently w.th reference fi. 



10A to IOC 



5 A further partial 3D reconstruct is then generated b, mo ^camera CM to a 

• o nt CMA such that the object lies in a region of overlap 
new viewpoint CMA sucn m<u remains in 

a rotation 6 about the y axis (resultm 0 a 
rotation * about the * axis (resulting .n new axes , , y and ^) 
rotation k about the axis, followed by translate > " ^ ^ ^ 
resulting axes. Therotation k about the z> ax.s w,U ,n many cases be z 

35 in Figure 1. 
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. . „„ ima „ e at lhe viewpoint CMA the camera is moved to a new 
After ^ r^Z ln^ U acquirer, and displayed (the orientation of the 
positron 1A and a further rma q ft ^ panial 

camera being adjusKd ,» rem , . < -~ ^ 4 ta . _ 

3D reconstruction of object J „;„ u rM and CM'. 

a„a,ogous to that described above in connection wuh vtewpomts CM 

„ •„;„ of the x' V' z' coordinate system is shifted to the 
„ a is assumed that the ongtn of the * ^ x 

optica, centre of the camera at v.ewpom, CMA 

Y z then the relationship between any pent XYZ in the 

sy 's,em and the same point xyz in the *yx coordinate system . gtven by. 



15 



x" 


c 


Y 




Z 





-sin6 



jjy 1 

-cos6sin<J» cosOcos* 



X 

y 



+ 



AX' 

AY 

AZ 



The terms in the 3x3 matrix can be shown to be the cos.n*s = ^ 
f «„. VY7 and xvz coordinate systems (see eg Boas/wame^ 

Is Pub J ohnW, 1 ey & So„, 2 nd B1 npp437a„d 4 3S,. 

According the pania, 3D reconstructions can be t— to a _ 
coordinate system and e,ongate*compressed along « * ee a ^ ^ 

discrepancies between them in their reg,o„ of overlap 
complete reconstruction of the object 3. 

5 „ the present embodiment this process is earned out ~ * 

displaying the partia, 3D - * 

elongation/compression along three axes, 

reference to Figure 19. 

10 i , D reconstructions are combined without user 

ln another ^^^^^ Z algorithm. This algorithm is publicly 
intervention using the Iterative Closest Pom B ^ 
available and therefore it is not necessary to describe i J of 

35 — s two surface ^«^^^^^. —e the 
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pairs of points is found. The repositioning step is then repeated. Other methods of 
orrelating 3D surface regions are disclosed in our OB 2.292.605B. In accor nee 
with one aspect of the present invention, a scaiing factor or other vanable ■ 
generated either by the user or iterative!, under software control ,n order to adjust 
. the relative sizes of the partial 3D reconstructions to ensure they can be f.tted 
together into a self-consistent overall surface description of the object. 

Returning to the description of the embodiment of Hgures 1 and 2, further partial 3D 
relations can be derived similarly from the other sides of the object and 
.0 combined with each other and/or with the existing combmauon of parttal 3D 
reconstructions until a complete 3D representation of the object is achteved. 

The method is summarised in Figure 2. 

15 Overlapping images are captured (step SK», pairs of points (or larger features eg 
Hnes or areas) are correlated (step S20) and a, least the approbate camera 
movement be ween the viewpoints is determined (step S30). .n the preferreo 
Z Lent mis is determined with a high degree of accuracy by processing the two 
,„ ima.es Partial 3D reconstructions of the object surface are then generated m a 
2: ated 3D space ustng the computer 4 to process the correlated pairs of pom s 
and camera movement (step S40, and these are combined, preferab.y .nteracttve^ 
on screen by the user to give a consistent but possibiy distorted <^°« °J 
elongated) 3D representation (step S50). Optionally, this is distorted or und.storted 
25 by applying appropriate compression or elongatton (step S60). 

,„ an alternative embodiment, steps S30 and S40 can be combined ta a matrix 
processing method, eg using the commercial* available INTERSECT program 
produced by 3D Construction Inc. 

30 

S<ep S40 will now be described in more detail with reference to Figure 3. 

R „ure 3 shows the object 3 in the field of view of the camera CM a. positions CM 
„ an°d CM' Pnncipal ray .ines from points Qa, Qb and Qc on the surface of the object 
" pal Trou-h oplal centres OC and Oc' of the camera in the respective posmons 
CM T< CM' and are each imaged on the image plane. The pair of po.uts thus 
foled on the image plane from each of points Qa, Qb and Qc (and many other 
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points, not shown) are responding points and can be corded by known 

algorithms. 

According i. wi.. be appreciated that if vir.ua! projectors prl and P r2 with the « 
. o^ focal >e»g.h and other optica. Caustics as *. camera at 
' Tspective viewpoints were placed a. positions CM and CM then they would 
pro ec. ray iines into vir.ua, 3D space from the corrected pairs of potn, wine 
Lid intersect a. the true ,ocatio„s of the corresponds po,n.s Qa Qb, Qc ^.d 
ai, other points on the object surface corresponding to a corre late patr of tmag 
0 pits By a virtua, projector is mean, any operator on .he image whtch behaves as 
an optical projector of .he image. A sutable software routine for processmg the 
image in the reouired manner couid be written without difficuity be persons slalled 
in the art. 

15 Although .he projector are shown in Hgure 3 with .he same orien.a,ion in .he 
re^Tce frarn! of .he objec. (corresponding .0 .he common camera orient^) thts 
mereiy a preferred fea,ure which enabies .he vector V to be determtned more 
asuy by the image processing method disclosed in Figure 4. In pnneiple the above 
, n aclsis is also applicable to virtual projectors of different or.entat,ons, 
„1 s on n. to respective, different camera orientation, As noted above, the 
"enLions 1 be determined by an tnertia, sensor ^ — '"^ 
similar ,0 that disclosed in our above-mentioned UK patent OB 2.292.605B. 



25 



30 



35 



If .he vecor V joining .he o P .ical centres of the camera/projector lenses m pos.tton 
CM and CM' is extended and projector pr2 is moved along ,h,s vector w,.h tts 
orientation unchanged (eg .o posi.ion pr2' as shown) and .he same ray Imes are 
projec.ed from i. .hen .hese will s.ill intersec, the ray lines from projector prl but at 
different posi.ions in simula.ed 3D space. Fo, example .he intersects a. Q* Qb 
and Qc will be replaced by intersections a, Qa', Qb' and QC respecvely. Th,s ,s 
because a y ray 1 ne from Op' parallel to a ray line from Op wil. lie in the same 
p Ln as a ray , ne from 0 C intersecting .he ray line from Op and .herefore mtersec, 
Cray line from Oc For exampie line Op'Qa' lies in .he same piane as mangle 
OcOc'Qa and will therefore intersect line O C 0a, in this case at Qa . 

Hence a scaled representation 3/3' can be generated by any pair of virtual projectors 
onTe^tor V, if V is paralle, to the Hue joining the optica. cen,res o, .he camera lens 
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have .he same orients) as the camera. Th,s orienutio „ 
automatically in software. 

. „• „,v a 3D representation of the object can be generated once the direction of 
0 ACC :r 'v Low F gnr. 3 will be referred to again in connection with a very 
I-^r ml'd applica^e to the virtual projectors assocated with the proctor- 
camera aspects of the invention. 

a Hgure 4 —s one delation ^^^^Z - 
understanding, the camera (having a lens wit p u M p oints 

stationary and the object 3 is considered to move to po, ■ «3-« . s 

P1 , P2 and P3 are ^on - * - * *• 

and as points pi . p2 and p3 wnen j unclian „ ed between the two 

„ orientation of the camera relative to » ^ ^ ,,,, p „, ( „ , , „ 

positions, then the pactions "^^^ , meet a , a common point 
1 in this illustration but in general will be mucn iar 0 er, 
VP, which is somewhat altin to the vanishing point in perspective drawing. 

» It wil, be seen tha, the Hue fining VP to the optica, 

the locus of the optical centre of the camera (or desired locus of 

relative to the object. 

, ,j r«~c ti 19 L3 .Ln connecting the 
,n general, if the camera is hand-held, the hnes LI, L2, 13. 

30 correlated points will no, meet a, a common point, ^Z^Z camera can 

be varied about the X, Y and Z axes by ^ 

t r ,u 19 13 Lnon screen and tne seconu iHm 0 r 
displaying the lines LI. L2. L3...LD o determin ed either visually 

» ="bT^ r — - - 

software routine, on!y when the necessary convergence is achieved. 
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,f escape G is used .hen its 3-axis orientation signals can be used ,o maintain the 
orientation of the camera between the two viewpoints. 

should be noted that if the camera is considered to be moving and the object to be 
' rl the movement of the camera from its oHgin* P™ 1 

position on the image plane (ana ima„e, o oosition and 

u ,• I i I 2 L3 Ln in the image plane of the camera in its first position 

: t L"«ti'n= point of intersection VP to the optica, centre of the camera 

coordinate frame of the camera at its first position. 

■«„t of the above method, a moving image or a rapid succession of still 
I„ a vanant of the above m ^ ^ 

change in orientation) of the camera. 

„ should be noted ,a, when there is little or no — of the camera reiative to 
'5 .u KWt in the depth (Z) direction then lines LI, L2, L3...U1 win h 

Ito aid VP J> beat infinity or too far away to be determined with accuracy. 

TM s specia, case is illustrated in Plg^s 5 - .Pom, A B and C . 
objects in its initial position are imaged by lens L tnan 

rSrZS"^ - — be ~ ~ ~ 

explained beiow, it will be assumed initially that points A, B and C P 
35 which is substantially parallel to the image plane 1. 

H should be noted that the images abc and ,„V would also b. , o*ai„ed from a 
smaller object of the same shape correspondingly nearer the camera, as illustrated by 
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faceAlBlCl. 



Referrino now to Fioure 6 which shows the object 3/3' (not its image) as seen by the 
Refemng now h a2b2C2 ^ ^ There Wlll 

camera, vanous possibl ^ ^ ABC; ^ ^ ^ of clarity only the 

be a cont nuous range of possible sizes ior ia . p 

above three are shown. However t he possible faces a,, have a common centro,d P. 

* v illustrated bv face A'B'C the centroid 
When t he object moves to a new posU,o 3 ,„ u s„ * ^ ^ ^ 

will move to a new position Q and the line rv. r 
1 1 (Fi-ure 5) will represent the direction of movement irrespective of the true 
the object. ,. fact this will remain true even if there is some 
Zl about the Z axis (such that the lines AA\ BB' and CC are not parallel, 

However if the camera is rotated at the second position about the Z axis to ensure 
However ,1 the ^ ^ aboye ana , ys , s 

that me ^^"^^ C do no, He in a p.ane parallel to the image 
holds true even if he poina A B aibc ^ ^ & ^ 

3 P and the corresponding centroid of Al'B'C after movement of the object o 
!tn 3- is similarly displaced from centroid Q. However the line joining these 
r — ^ b/pale, to line PQ and will therefore correctly indicate the 
direction of movement of the object relative to the camera. 

* Even if there is some movement of the camera in the Z direction such .ha, ^ines AAj 
BB' and CC are not parallel but converge to a distant point, line PQ »,,! 
correctly represent the direction of movement of the object 3. 

Thus the above method overtaps to some extent with the method of Figure 4. 

30 

The overall method of determining the direction of movement of the camera is 
ILrated !n Figure 7. A firs, image is captured (step SI.) and the camera is moved 
! Z ^si«°n with the object still in „ field of view and the firs, ,mage ■ 
35 1 Jyed „ screen 5 (Figure 1). superimposed on the instantaneous image s n by 
rdera a, the new position (step S,2, Correspond*, , potn s are correla^ 
either by eye (ie by the user) or by a suitable software rou .ne^ep S13 . On, y a 
smaU number of points need to be corrected a. this stage, eg 100 or fewer, 
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depending on the processing speed and the accuracy required. 

If it appears to the user that there has been movement in the Z direction the method 
branches to step S14 at which the orientation of the camera is adjusted about the X, 
j Y and Z axes until the correlated points converge to a common - vanishing pent VP 
(Figure 4). At this stage the second image is captured and stored in memory. 

In step S15 a line is projected from the point VP through the optical centre of the 
camera lens to find the locus V (Figure 4) and with the aid of this .nformation, 
'0 furth er correlation of the images is performed and a partial 3D reconstructs „ 
generated by the moethod illustrated in Figure 3 (step S17). 

Preferably, step S 15 also involves or is preceded by the selection of a camera model 
(e- from a list delayed on screen) by the user. The computer 4 is programmed to 
15 Jre a list of camera models Ml. M2....Mn in association with the parameter . o 
each model needed for the calculation of the required projects and other 
processing/The following parameters may be stored: 

20 Camera model Mn: 

pixel size in x direction 
pixel size in y direction 
focal length 
25 x dimension of film plane (in pixels) 
y dimesnion of film plane (in pixels). 

,f it appears to ,he user (eg as a resuU of a faUure to find a. reasonable 
.•vanishing point" VP) .ha. .here has no. been significan. movemen. .n ,he Z 
30 Z L JL .he movemen. of .he objec. is assumed .o be paraHei .o me .mage 
p,a„e and .he .ine PQ is found from .he images of a group of eg <hree or more po,n.s 
(no. necessarily come.) preferabiy lying in or Cose .o a plane paraile, .o U,e .mage 
plane (step S16) before proceeding to step S17. 

35 If „„y because mos, digi.a. cameras currency on .he -« have a limi.ed 
resoJion (eg !024 x 768 pixe,, «« x «0 pixe.s or even iess) *erej~ 
ineviubly be some error in .he determinaUon of .he camera movemen. and hence 
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the relative placement and orientation of the virtual projectors PR1 and PR2 in 
Figure 3. The effect of this error is illustrated in Figure 8. 

Rcure 8 is a ray diagram orthogonal to the image plane showing the imaging of a 
lir Ie of points PI. P2 and P3 on the surface of the object 3 by a camera at posmons 
CM and' CM'. For the safce of clarity the lens L * shown only at the first pos^n 
CM. It will be noted that the orientation of the camera is the same at the two 
positions. 

1 Virtual projectors prl and pr2 would correct* reconstruct the object as 
reconstruction 30 if .oca.ed at the above positions with the above onen.at.on If 
projector pr2 were moved to position pr2' along the direction of movement V 
£^4) of the cameras then the resulting reconstruction 30' would stil, show the 

5 pofnts (P,-. «• and «') on a straight line with no curvature of ' fl*«J. correct 
scaling in me iatera, direction (ie the length ratio P1P2/P2P3 = PI P2 /P2 P3 ). 

If the projector is successively rotated to a new positions P r2" and p^Mhen 
successively greater curvature is introduced into the reconstructs 30 or 30 as 
» own. The rdationship between the above curvature of f.e.d of the reconstn.ct.on 
30 and the angu.ar error in the orientation of the second virtual projector relauve to 
the f.rst is considered below in relation to Figures 10A to IOC. 

However, referring to Figure 9, in which the respective ray lines from the collated 
* points of the images projected by projectors prl amd pr2 « 

orthogonal, it can be seen that an error in the relative orientation of the projectors, 
resulting in the projection of the second image from projector P r2 resu s .„ a 
negligible curvature of Held: line 30' defined by the .ntersecUon of .he r y hnes 
from V with the corresponding ray lines from factor prl .s substanually 
30 straioht. like line 30 which is defined by the corresponding intersections of the ray 
lines from correctly co-oriented projectors prl and pr2. Accordingly, .s a preferred 
ZL of the invention that the camera positions are so chosen that the angle 
subtended by the ray lines from a pair of correlated points at the-r , ntersecUon a. 
35 their corresponding point a, the object surface is substanfally 90 degrees, g <*> 
deorees * 30 degrees, preferably 90 degrees ± 20 degrees, most preferably 90 
delrees * 10 degrees. This minimises the correction required due to curvature of 



field. 
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The re remains the latera, distortion o f -« * 

the ratios P.P^ K ^^J^X 1 ^^^ 
30- U ^r-^; 3 ^:,: p int I are shifts so as to to be 
of projector PR1, as shown ■„ figure 9 and a 1 p ^ 
intercepted by d» ra y Hnes from ^ < then . ^ ^ ^ 
shifted points and .he ray hnes from prl w. U be ^ y ^ 

aefined by ,he correspond „g Po.n. o n 30 a r^ J ^ ^ 
dilution of pom* along . 0C w 1 * ^ ^ ^ ^ ^ ^ 

r X nTi; 1" Hnes a, their —on at their corresponding point a, 

the object surface. 

According, distortion paraUe, to the image piane can be com *ted for^r 

Accoruiu c i)) rprnnstruction about an axis 

15 deliberately applied) by rotating the part, 3D — ^ 

perpendicular to the image plane of a r 3 ^^^ ^oo-rucdoo «o lieon 
reconstruction whilst constraining the points m the partial 3D recons 
their ray lines from that projector. 

20 . _ „r r.*»lH resulting from angular 

x=- . c lftA to 10C illustrate the curvature of field resumn e 

CN of the actuai reconstruct,™ 30 . shown ^ ^ ^ 

The radins of — K - '^^2^^^ 

a , ( ,he reconstruction 30 of, the object surface ,s J„ ^ 

angle of 90 degrees and hence the degree of curvature ,s very m 
would normally be obtained in practice. 

35 Hence the above method of the i— ^^^^ £ £ 
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The corrections noted above can be applied a, any stage during the generation or 
fitting together of the partial 3D reconstructions. 

, I. should be noted however that only a limited misalignment of the virtual projectors 
rt a d pr2 (F.gure 3, is permissible. Referring to Figure 3. a„ the pnnctpa! ray 
l p „ ectL from the correlated points win intersect only if the promts have 
the correct orientation. The latitude in orientation arises only from the fmue 
resdution of the cameras) which implies that the ray lines from the projectors have 

.0 a finite thickness o, do not intersect exactly. Thus a point on * — -» 
can be found by determining the midpoint of the shortest hne jo.nmg the ray hnes 
ZZL respective projectors, provided that the length of this shortest l.ne does no, 
exceed a predefined limit corresponding to the resolution of the cameras). 

15 A projector-camera embodiment closely analagous to the camera-camera 
embodiment of Rgure 1 will now be described with reference to F.gure 12. 



S a 



20 



25 



30 



Referring to Figure 12, the apparatus comprises a personal computer 4 (e 

p f„ m ® PC) havina conventional CPU, ROM, RAM and a hard drive and aframe 
Pentium^ r^) naving ^» vtu „ video output 

crabber connection at an input port to a digital camera CM as well as a P 
irt connected to a screen 5 and conventional input ports connected to a keyboard 
[Use 6 or other pointing dev.ee. The hard drive is loaded w.h conventual 
operating system such as Windows®95 and software: 

a) to display images acquired by the camera CM; 

b) to generate correspondences between overlapping regions of images input from 
the camera CM; 

c) to derive from the acquired images the baseline joining the perspective centres of 

the camera and projector; 

d, to proiec. images acquired by the camera into a simuiated 3D space from virtual 
projeli located on the baseHne at a separation seated (w.th the keyboard or 
pointing device) by the user or determined (thereby creattng a pamai 3D 

reconstruction); 




PCT/GB99/01556 



WO 99/60525 



29 



5 



10 



.) ,o scale ,he partial 3D reconstruction along one or more axes and combine such 
partial 3D reconstructions as illustrated in Figures 17, 18 and 19, and 

0 to determine the separation of the perspective centres of the camera and projector 
L. the baseline from further correlations of object images and cal.brat.on .mages, 
and Lnce derive an accurate partial 3D reconstruction of the object surface. 

Additionally the software is preferably arranged to correct the images for distortion 
to curvature of fietd of the camera and projector optics before they are 
processed as described above, either during an initial calibration procedure or as 
p rt of a ray bundle adjustment process during the processing of the object and 
alibratic, .mage(s). Suitable correction and calibration procedures are 
Tsai in -An Efficten. and Accurate Camera Calibrate Technique for 3D Machme 
■ Vision" Proc IEEE CVPF (.986, pp 36«74 (supra) and w„l no. be descnbed 
further. 

The camera a Pulnix M-9701 progressive scan digiial camera, is shown mounted at 
o ZZ support frame F on eg a ball and socket mounting and a slide projector 
° PR is shown securely moun.ed on .he other end of the frame, S.ide projector PR >s 
^o ded with a speckle pattern siide S and is arranged to project me result.ng 
Ipeckle partem onto the surface of region R of an object 3 wh.ch ,s ,n .he field of 
view of camera CM. 

The intrinsic camera parameters are initially determined by acquiring .mages of a 
Reference plate (no. shown) in Known positions. The reference plate ,s planar and has 
17Z of printed blobs of uniform and Known spacing. The follow.ng parameters 
2 de.erm.oed and are .nerefore assumed ,o be known in the subsequent descnpUon. 

30 

i) focal length of the camera 

ii) distortion parameters of the lenses of the camera and projector 

iii) scale factor 

35 iv) image coordinates of principal point. 

Additional* the pixel size (determined by the camera manucturer) is assumed to be 

known. 
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Optionally, the following extrinsic camera parameters are determined: 



a) camera location 

b) camera orientation. 



Alternatively .he camera location and option can be taken to def.ne the 
llate system relative to which the object surface coordinates are determined. 

.0 Refers now to Figure 2, the camera CM is shown with its perspective centre OC 
LiTd "on a baseline vector V and viewing (initially) a targe, surface T and 
sclent*) object 3. The (virtual) origin or perspective centre Op of projector PR 
o s on seile vector V and is defned by the optica, system of the project, 
elm risin. field lenses OL and condenser ienses CL. A point hgh. 
15 Z filament bulb iiiuminates siide S and directs a specie pattern onto (,n,«a„y) 
target surface T and (subsequently) the surface of object 3. 

The baseline vector V is found by the following procedure: 



20 



PirsUy ani m age,l(F,gure,4,of,he region of the surface of object 3— d 



yte p" S P— ™ is acqui.d and stored ,„ ^^^Zl 
a d an arbitrary group of a. leas, two spaced apart ^ 1 ^^^ D 
are seiected as points ,1 and ,2 in the .mage formed on the photodetector plane PD 
25 of the camera. The group of points ql and q2 .s stored. 

Secondly the object 3 is substituted by target surface T and an image 12 (Figure 3) of 
, h e. illuminated region of the target surface is acquired by camera CM. The posru 
Z of°.he target T re.ative to the camera are found by — g an^ma g 
30 of the target in the absence of any illumination from the projector, ufl.s ng a known 
pauem of bs BL formed on the periphery of the target. The image .2 ,s stored and 
a patch defined by its centra, point (eg Q„, Figure 3) of the firs ,n^ U . 
correlated with the corresponding point P„ of the second .mage 2 b se, cun 
, lundin. region R of initially 3 x 3 pixels and, by companng local radtomemc 
" drs.bu.ions by mens of the above-described ^^Ztl 

searching for the ^ in 

distorted with an affine geometric distortion (eg m the simp 
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fi„ur. 14 horizontally eionga.ec!). The correlated patch is expanded (up to a 
malum of » * 19 pixels) and the process is repeated. In thts tnanner the 
corresponding point P n is found. 
5 This process is repeated to find a ,ar g e number of pairs of * 

with the points in the group Ql, ^ (,ri D urc x ; 

resoiution. the !auer are no. necessarily centred on parttcular ptxe!s. 

10 .„ the following -eometric discussion the correspondences are treated for the sake of 
'Ip i^a ^.ated pairs of points out it should he noted that this does no tmply 
anything about their topography - in particular it does no. tmply that they 
corners or edges of the object, for example. 

15 .A ,h.orioin OP (perspective centre) of the projector PR will lie 

Referring to Ftgure ^£*J^, lhe 3D lora tions of these four points 
at the > m '™ om [he camera on whic h they ,ie, name.y p.Pl. 

;rr P t; wn and X z *. ..»= «<» *> - - »- ° cpiqi >t: 

, ^ mailable form the caHbrau on pr^ ant 1 the ^ » ^ 
12 and the line P2Q2 will lie in the plane Q C P2Q2 ie plane UCP J 
similarly available from the calibrate process and ^^^ ^ ^Z 
planes define a baseline vector V by their mtersection, winch passes throu e 
the perspective centre Op of the projector. 

" A pa— simple *ay ^^^^^ 

30 .v, h^eline vector V can be determined, though not the position of 

In this manner the baseline vecior v v. „ rt : nt , p and O 

a least squares estimate of vector V on the remainder of the vectors, 
with known statistical methods. 
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Th e aerivation of a three-dimensional -P— * <»< <*« 3 * * 

ray diagram of Figure 3. 

„ ■•rtnr PR are shown located on baseline vector V. A first 

The camera CM and * processing software in computer 4 

virtual projector prl is im plemented by the .«.. P ed . n ^ ^ 

and has the same optical charactensUcs as he «-(^ ^ ^ ^ 
calibration procedure). Image 11 (Figure 14) is p j 
in a3D space simulated by the image processing software. 

A second virtual projector P r2 is ^ 

software and preferably - *e same proje J a set of ra y 

(which is also represented m Figure 5). n respective physi cal projector 

- " - —a P ,eu or 

rays PQ and the ray lines are each laoeue riMCli bed with reference to 

,hl image U as founa in the image ^ _ fce 

Flgure ,4. U wil. be appreciatea .ha, .he .mage ^ Qp of the 

e q „a.ea wi.„. a se. of rays ong.na.mg from *W 
p roje c.or. Since UU town which ray ime from .he pro^ P P 

Qc... defining the surface. 

•„„ of R-ure 3 it has been assumed lha, .he relative positions of 
In the above a.scusston of F.gu e 3 , ^ ^ ^ 

10 the camera CM a„a and projector ™ ^ ^ or entered 
of the virtual projectors prl and pr2) are known, 
as a scaling variable by the user of the computer 4. 
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That the ray iines from the respective v.rtua, projectors - 

- 77, tlCte ^^1—:^. projector position p* 
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reconstruction of the object 3- -ill be different in s,ze but of the ann^hap. 
single scaling factor will be required to interconvert objects 3 and 3 . However ,, 
ly be convenient in pract.ce to provide for different horizontal and veruca, sea ng 
facLs (.. because of different horizontal and vertical magnincafons of the came a) 
l„d in I era. only one set of scaling factors will be consistent with fitting together 
a set of partial 3D surfaces of the same object acquired from different d.recnons. 

Accordingly the software ,n computer 4 is arranged to scale such acquired 3D 
tprtttatLs to enable them to be fitted together to form a self-cons.sten. overall 

Representation of the object. This aspect is described be.ow with reference to 

Figures 17 to 20. 

Before doing so however, an alternative calibration procedure wU, be described whh 
reference to Figure 15, which shows two planar calibration targets T. and T2 

5 (havin. periphery blobs or discs BL similar to target T of Ftgure 13) whose 
orisons (and preferably positions) relative to the camera axis system are Known, 
Z T1 --It of a photogrammetnc determination involving separately acqu.nng 
taages of them in the absence of any illumination from the projector, and processing 

„ the La-es in a procedure similar to tha, described above in connects w.th figure 
2 The P-pecuve centres Oc and OP of the camera and projector are also shown, 

,„ a frrst stage of the calibration procedure, target Tl is illuminated by the structured 
BH. from the projector and an image is acquired by the camera CM. figure 15 
* i^trates three points p., P 2 and P 3 a, wh.ch the structured light ,mp,nges on target 
Tl. These (and many other points, not shown) will be imaged by the camera CM. 

In a second stage of the calibration procedure, target Tl is removed and target T2 is 
iUuminated by the structured ,ight from the projector. An tmage , acqu,^ b, £ 
30 camera CM. The three points PI, P2 and P3 correspondtng to P"^*^£ 
are found by correlating the newly acquired image of the project.™ of the structured 
^Lon on target T2 with the previously acquired image of the correspond 
projection on Tl by the procedure described above with reference to Rgure 13. 

M figure 15 illustrates further the relationship between the positions of two calibration 
J„ets Tl and T2 and the perspective centre 0 P of the projector PR (the came* .CM 
1 assumed fixed on the basehne vector V). A pair of points PI and P2 on urge. 



10 



PCT/GB99/01556 
WO 99/60525 



34 

Tl form image points pi and P 2 respectively on the photodetector array PD of 
camera CM and (in a subsequent step following the removal of target Tl) a pair of 
points P3 and P4 on target T2 which are correlated with PI and P2 respectively 
form image points P 3 and p4 respectively on photo detector array PD. 

Accordingly in a third stage the pencil of rays formed by corresponding points on 
taraets Tl and T2 (eg PI and P3; P2 and P4) is constructed to find the position of the 
perspective centre Op of the projector. In practice the rays will not intersect at a 
point but a best estimate can be found from a least squares algorithm. 

It will be appreciated that other calibration procedures are possible. For example the 
camera could be calibrated by the Tsai method (Roger Y Tsai, IEEE Journal of 
Robotics and Automation RA-3. No 4, August 1987 p 323 - see also references cited 
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therein). 
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